An important step in producing a high quality software program is to test the program before it is generally released. A poorly tested program increases production costs and time, and decreases user satisfaction. As programs become more complex, it becomes more difficult to ensure that a program has been adequately tested. An adequate test exercises as much functionality of the program as possible with input data that are representative of real data processing problems. The testing should reveal if the program has any flaws in its design.
One way of verifying program testing is called "test coverage analysis." Test coverage analysis measures the scope and quality of a test by determining how much of the program was exercised during the test. Typically, test coverage analysis requires three major steps. First, the program to be tested is modified so that its execution can be monitored. The modifications generally entail adding instructions to the program. The "monitoring" instructions will divert the execution flow to monitoring procedures which can record the fact that a portion of the monitored program has been executed.
Normally, code modification, or "instrumentation" as this is sometimes known, is done in two ways. Either the source code is modified and the modified program is compiled as usual, or an executable image of the program is modified.
In either case, during a second step, the modified code is loaded for execution into a memory of a computer system. During execution of the modified program, the execution flow is intercepted at the instrumented instructions. The monitoring procedures can then gather and record test coverage data which are significant of the operation of the program.
During a final step, the test coverage data are analyzed to determine the behavior of the program in response to a particular set of input data. During this step, it can be determined if all portions of the program were adequately tested. If this is not the case, then the input test data can be modified, and the testing cycle can be repeated until a significant portion of the program is tested.
Conventionally, test coverage data have been presented as an annotated listing of the source code. Table 1 shows such an example listing.
TABLE 1 ______________________________________ 125 ++ void p(int I, int j) { 126 ++ if (I==1) { 127 -- I=j=0; 128 ++ } else if (j==1) { 129 ++ k=0; 130 } else { 131 -- q( ); 132 ++ } 125 ++ } ______________________________________
In the table, each source code line is preceded by a line number and a code. The code "++" indicates that the line was executed during the test. The code "--" indicates that the line was not executed. The uncoded lines do not have any equivalent expression in the machine executable image.
In this example listing, it can be seen that during the test, "I==1" on line 126 always failed because line 127 was never executed. The test for "j==1" on line 128 always succeeded because line 129 was executed, but line 131 was not.
This manner for presenting test coverage data is relatively easy to understand for simple small programs. However, real programs may include thousands of lines of code distributed over possibly hundreds of source code files. Program developers rarely read source files serially from start to end, but this is just what is demanded by this type of presentation.
More importantly, this type of presentation does not readily distinguish what is important, from what can be ignored. In practice, test coverage of an entire program can never be complete. For various reasons, such as program complexity, and many possible input states, only about 80% to 90% of coverage is typically attainable. The remaining unexercised code can be scattered over tens or hundreds of source files. A serial perusal of the entire analysis presentation to uncover potential design flaws can consume extra time, and introduce errors.
A more useful presentation of test coverage data would take into consideration the logical structure of the program, and quickly suggest how to change the test to cause more of the code to be executed. Moreover, the test coverage information would be presented to a user in a form which is not necessarily serially organized according to the order of the instructions. In contrast to the linear listing above, a useful presentation would focus on the significant unexecuted portions of the program so testing could be improved.